{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第13课：动手制作自己的简易聊天机器人"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自动问答主要研究的内容和关键科学问题如下：\n",
    "\n",
    "问句理解：给定用户问题，自动问答首先需要理解用户所提问题。用户问句的语义理解包含词法分析、句法分析、语义分析等多项关键技术，需要从文本的多个维度理解其中包含的语义内容。\n",
    "\n",
    "文本信息抽取：自动问答系统需要在已有语料库、知识库或问答库中匹配相关的信息，并抽取出相应的答案。\n",
    "\n",
    "知识推理：自动问答中，由于语料库、知识库和问答库本身的覆盖度有限，并不是所有问题都能直接找到答案。这就需要在已有的知识体系中，通过知识推理的手段获取这些隐含的答案。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /Users/zhangjianfeng/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/zhangjianfeng/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/zhangjianfeng/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'ChatBot' object has no attribute 'set_trainer'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-07828e3272cd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mchatterbot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainers\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mListTrainer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mChinese_bot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mChatBot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Training demo\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#创建一个新的实例\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mChinese_bot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_trainer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mListTrainer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m Chinese_bot.train([\n\u001b[1;32m      6\u001b[0m     \u001b[0;34m'亲，在吗？'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'ChatBot' object has no attribute 'set_trainer'"
     ]
    }
   ],
   "source": [
    "from chatterbot import ChatBot\n",
    "from chatterbot.trainers import ListTrainer\n",
    "Chinese_bot = ChatBot(\"Training demo\") #创建一个新的实例\n",
    "Chinese_bot.set_trainer(ListTrainer)\n",
    "Chinese_bot.train([\n",
    "    '亲，在吗？',\n",
    "    '亲，在呢',\n",
    "    '这件衣服的号码大小标准吗？',\n",
    "    '亲，标准呢，请放心下单吧。',\n",
    "    '有红色的吗？',\n",
    "    '有呢，目前有白红蓝3种色调。',\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 测试一下\n",
    "question = '亲，在吗'\n",
    "print(question)\n",
    "response = Chinese_bot.get_response(question)\n",
    "print(response)\n",
    "print(\"\\n\")\n",
    "question = '有红色的吗？'\n",
    "print(question)\n",
    "response = Chinese_bot.get_response(question)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lines = open(\"../data/13/QQ.txt\",\"r\",encoding='gbk').readlines()\n",
    "sec = [ line.strip() for line in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['那你一定没看过编译原理',\n",
       " '上过最恶心的专业课',\n",
       " '没有之一',\n",
       " '觉得对现在工作没用',\n",
       " '不会一点影响都没',\n",
       " '学习不是为了有用而学',\n",
       " '有谁搞过网络拓扑自动发现啊',\n",
       " '学习是因为学习使我快乐',\n",
       " '不然我早就跑了',\n",
       " '真心羡慕那些自学能力强的人',\n",
       " '这是那个做了宾果消消乐的那个单位的校招宣传海报',\n",
       " '但是',\n",
       " '在哪里啊',\n",
       " '叫什么柠檬啥啥公司、记不清了',\n",
       " '北京',\n",
       " '是基本工资么',\n",
       " '这个薪资',\n",
       " '税后',\n",
       " '我总觉得人家大专出去的',\n",
       " '北京不算高',\n",
       " '也有这么多',\n",
       " '我05年的时候去北京就是9000多',\n",
       " '税后',\n",
       " '6',\n",
       " '毕竟大佬儿',\n",
       " '[表情]还是基本工资',\n",
       " '05年的9000+ 和现在的1w',\n",
       " '那简直不是一个概念',\n",
       " '现在研究生都不值钱',\n",
       " '努努力在北京买房啊',\n",
       " '那我这个本科生能出去干什么..',\n",
       " '双非..还是 第三批次本科',\n",
       " '挂机等死',\n",
       " '大专',\n",
       " '那死定了啊',\n",
       " '6月份满2年',\n",
       " '兄弟',\n",
       " '挂机、不如浪一波',\n",
       " '小学呢',\n",
       " '现在税后8k',\n",
       " '小学毕业进的培训班',\n",
       " '现在10多年经验了',\n",
       " '啥都会',\n",
       " '初中就去搞代码？怕不是猝死哦',\n",
       " '就有人吹牛逼，小学开始写代码',\n",
       " '写汇编',\n",
       " '可牛逼了',\n",
       " '高中就赚外快了',\n",
       " '大学就创业了，当cto',\n",
       " '大一时我特么还只会hello world',\n",
       " '大二呢、',\n",
       " '大二的时候我去参加了一个算法比赛',\n",
       " '个人见解啊。现在不管做java还是大数据，项目需求一定要和智能，决策这种概念搭点边。',\n",
       " '项目价值才大。',\n",
       " '如果还是一味做老的数据操作，增删改查，是不值钱。',\n",
       " '拿了省二奖',\n",
       " '良_datamining_深圳 会个for循环了啥的',\n",
       " '大三看hadoop文档',\n",
       " '发现都是天书',\n",
       " '什么都不会',\n",
       " '只会吹水',\n",
       " '大学基本上都在学数学证明',\n",
       " '可爽了',\n",
       " '数学专业？',\n",
       " '嗯',\n",
       " '那挺好的',\n",
       " '我现在想回学校学数学',\n",
       " '本科中工作你试试',\n",
       " '计算机有数学底子太棒了',\n",
       " '看着各种算法好累',\n",
       " '数学转计算机转挺好转的',\n",
       " '有时候根本看不懂',\n",
       " '算然这么说',\n",
       " '无法理解',\n",
       " '但是数学专业的找工作可酸爽了',\n",
       " '做小学数学老师啊',\n",
       " '计算机专业最好就业',\n",
       " '能找到好工作的基本不上课的',\n",
       " '你们天天上班的时候吹水',\n",
       " '不上课写代码找项目的',\n",
       " '老板不会骂人吗',\n",
       " '我姐同济的数学专业啊',\n",
       " '好找公祖哦啊',\n",
       " '那是自学的',\n",
       " '都是屁话，谁教你们的',\n",
       " '工作',\n",
       " '别人背后有多努力，你根本不知道',\n",
       " '同济是出 高数书的',\n",
       " '对啊，别人每天熬夜看书而你是在熬夜打游戏',\n",
       " '统计 土木 建筑之类的非常强',\n",
       " '同济',\n",
       " '数学专业开始没有找it工作的想法',\n",
       " '不可能的',\n",
       " '数学可以读研转IT啊',\n",
       " '那些自学的是学习不好的大部分',\n",
       " '所以数学的只能读研',\n",
       " '好多专业课理学院也开的',\n",
       " '要不找个好工作，基本学习不好的',\n",
       " '什么计算机网络 操作系统 啥的',\n",
       " '好像有的数学专业也会学的',\n",
       " '大三才想找工作，才开始准备计算机',\n",
       " '大部分人这样，不排除有大牛',\n",
       " '没有老师引导不可能自学计算机的',\n",
       " '老师只会教你怎么保研读研',\n",
       " '不太清楚、本科研究生都是计科',\n",
       " '我室友专业第一试过找工作，被百度鄙视',\n",
       " '他写代码算法也是数着的，只是没项目经验',\n",
       " '被鄙视不值得郁闷，反而该开心，这就是眼界的提升',\n",
       " '我面试腾讯，面试官就鄙视我说你只会算法啊',\n",
       " '一个个都是牛逼',\n",
       " '我能说啥',\n",
       " '嗯,只会算法本科很难做什么',\n",
       " '我都好奇我简历写的很清楚',\n",
       " '为啥让我去面试的',\n",
       " '腾讯就C++要熟',\n",
       " '是的',\n",
       " '看不上我还叫我面试',\n",
       " '只会算法不会开发语言吗',\n",
       " '浪费高铁票',\n",
       " 'java会基础啊',\n",
       " '这你肯定要被鄙视',\n",
       " '肯定会一门语言实现算法',\n",
       " '最好有大厂实习经验',\n",
       " '只会基础不行啊',\n",
       " '这种是学数学的吧',\n",
       " '找工作再学习，大牛可以做到学得快',\n",
       " '我们就是数学专业',\n",
       " '做算法很好啊',\n",
       " '我当时就是清华实验室做事啊',\n",
       " '刚部署的环境报这个错什么意思？访问HFDS报的',\n",
       " '本科算法学的不精，没项目经验是不行的',\n",
       " '只会算法那就学啊，算法都行，代码有什么难的。',\n",
       " '自己给自己设限了吧',\n",
       " '学的语言多了',\n",
       " '学数学不怎么好找计算机编程开发的工作哦',\n",
       " '你们看看就知道了',\n",
       " '学数学适合去大企业',\n",
       " '发现掌握一门语言挺简单的',\n",
       " '本科数学的找工作的',\n",
       " '有件事不明白',\n",
       " '做着搞商业智能的',\n",
       " '大佬不是很忙吗',\n",
       " '那只是掌握',\n",
       " '为什么大佬你一直在聊天',\n",
       " '需要你精通',\n",
       " 'MrGao因为我们都是虾米',\n",
       " '谁是大佬',\n",
       " '你说说Java体系有多么庞大',\n",
       " '离实际工作要求还差十万八千里',\n",
       " '我请假2小时去看中医',\n",
       " '金思图像识别年薪38万的大佬',\n",
       " '看他一直在聊天',\n",
       " '庞大你就别停啊',\n",
       " '18级华为才给38万是不是少了哦',\n",
       " '我们专业的大部分人虽然说只会算法，但是达不到精通，因为我们学这个，反正是菜鸟就对了',\n",
       " 'yyy是',\n",
       " '18级级别起码是个部门经理级别了吧',\n",
       " '我就是部门经理啊',\n",
       " '嗯！',\n",
       " '新部门待遇不高',\n",
       " '我知道的一个一个部门经理年薪60w+',\n",
       " '也不是他的核心业务',\n",
       " '差不多啊',\n",
       " '我们少点口头说50W左右',\n",
       " '我估计要打个7折吧',\n",
       " '不打折啊',\n",
       " '别个在华为很多年',\n",
       " '我自己猜测的',\n",
       " '10年应该有了',\n",
       " '我工作14年才18级',\n",
       " '比较笨啊',\n",
       " '年轻有为啊',\n",
       " '我不年轻',\n",
       " '35岁了',\n",
       " '35就是中层领导了',\n",
       " '不错了',\n",
       " '一个一个的都是大佬',\n",
       " '动不动都是年薪50W-60W的',\n",
       " '我年薪15W',\n",
       " '蜡笔没小新撤回了一条消息',\n",
       " '这么感兴趣、直接去咨询好了',\n",
       " '收好不谢',\n",
       " '这是老的吧',\n",
       " '一个个都是名校',\n",
       " '就没有小学毕业的嘛',\n",
       " '有还只是个宝宝的',\n",
       " '我小学毕业',\n",
       " '有人知道spark怎么读取hive数据吗',\n",
       " '简单的读知道',\n",
       " '就是简单的连接读取',\n",
       " '看我之前出的视频可以',\n",
       " '哪里有视频啊',\n",
       " '我博客有代码',\n",
       " '根本没有啊',\n",
       " '群公告有我们天亮教育的公开课视频',\n",
       " '我新手',\n",
       " '东部伦少求地址',\n",
       " '我新手，其他人就不用看了',\n",
       " '学生吧',\n",
       " '为啥这么说，毕业年快两年了，只是才开始用spark而已',\n",
       " '前端javapythonc#都在工作中用过',\n",
       " '华中科技大学的博士',\n",
       " '不行吧',\n",
       " 'oracle如何实时把新增数据发送给Kafka',\n",
       " '那个大学的都可以',\n",
       " '关键还是自己的能力OK',\n",
       " '有没有python的大神在，求教个问题...',\n",
       " '为啥这里点不出方法',\n",
       " '自己手写上去的不报错还能正确执行？',\n",
       " '编译器问题',\n",
       " '我用的pycharm还能这样，怎么能点出方法，要不感觉好别扭',\n",
       " '看下官方文档吧，调用的方法可能改了，编辑器识别不了',\n",
       " 'python连接hive是不是特麻烦',\n",
       " 'python交易起来特别方便',\n",
       " 'PY交易',\n",
       " '连接不来，现在只能用shell',\n",
       " '实时用户数量怎么做',\n",
       " '对大量数据，能不能有啥可以分析。然后得到个数学公司',\n",
       " '公式',\n",
       " '求大佬们指点',\n",
       " '大佬很忙',\n",
       " '在Spark中，加载一个文件，该文件的Rdd是会被分配到多个Worker中吗？',\n",
       " '那我退下了',\n",
       " '直接去看官方的例子',\n",
       " '[跟你妈说]',\n",
       " '执行spark-shell的时候报这个。。。有大神遇到过吗？？',\n",
       " '我的环境变量都配置了啊',\n",
       " '有人会解决sparkstreamingdriver端内存一直增长的问题吗a',\n",
       " 'hivemetastore无法启动有知道如何处理的朋友么',\n",
       " '下载好了mongodb,为何启动不了啊',\n",
       " '这个怎么解决',\n",
       " '报什么错',\n",
       " '图片里有',\n",
       " '就是这样',\n",
       " 'spark2.1中hivecontext换成了什么',\n",
       " 'sparksession',\n",
       " '有没sparksession的例子给我一个就是查询hive的',\n",
       " 'Hive好学吗？',\n",
       " '问一个大数据，大在哪里？',\n",
       " '好学啊',\n",
       " '大在数据啊',\n",
       " '这事为什么呀，全是null',\n",
       " '这类书每本一个动物',\n",
       " '这类书每本一种动物',\n",
       " '现在很少看纸质书了',\n",
       " '我就爱看纸质书',\n",
       " '我喜欢纸质的',\n",
       " '纸质书看起来有感觉',\n",
       " 'PDF每次都要跑去印刷出来',\n",
       " '佩服能看的进去书的',\n",
       " '喜欢看纸质书电子版的感觉就是看不下去但是搬家时候麻烦',\n",
       " 'Pig只是听说过有人搞过吗具体是啥东东为啥用Pig命名',\n",
       " '一看书就昏昏欲睡',\n",
       " '有没有对ES熟悉的',\n",
       " '急急急',\n",
       " '大神们',\n",
       " '说问题呀~',\n",
       " '王贵锋加入本群。',\n",
       " 'ES是啥',\n",
       " 'elasticsearch',\n",
       " '在用，但是一点也不熟',\n",
       " '不明觉厉',\n",
       " '不明白但是觉得很厉害',\n",
       " 'ES好东西',\n",
       " 'spark2.0和1.5兼容吗',\n",
       " '走人就走人',\n",
       " '你跟你妈说，今晚在同学家住',\n",
       " '冷少你在说啥呦',\n",
       " '哪位写过elasticsearch备份数据到hdfs集群，官方配置是写入8020端口，但是我的hdfs集群是基于yarn的，8020端口就根本没有开，不知道应该怎么玩啊',\n",
       " 'elasticsearch-hadoop',\n",
       " '用这个插件？',\n",
       " '问个问题。有人熟悉mongodb么？如果写java取mongodb中的数据类型，针对string类型但是值为null和字段类型是NULL的，怎么取？',\n",
       " '转换成json',\n",
       " '职位：项目实施经理',\n",
       " '月薪（天津华苑外高新区）：8~12K',\n",
       " '岗位职责：',\n",
       " '主导现场项目实施，编制实施文档，监控并定期汇报项目实施进度；',\n",
       " '收集、梳理客户需求，协助产品经理编写需求文档，对需求变更进行有效管理；',\n",
       " '监督指导实施工程师具体工作，确保实施计划落实，并对项目实施况绩效考核；',\n",
       " '熟悉公司产品，了解其主要的功能、安装维护方法及常见问题解决；',\n",
       " '记录实施工程师和客户反馈的问题和意见，及时反馈给产品经理，协助改进产品。',\n",
       " '岗位要求：',\n",
       " '计算机相关专业本科学历，三年以上工作经验；',\n",
       " '熟练使用办公软件及各种办公设备；',\n",
       " '工作主动，良好的人际沟通及协调能力，有较强的团队合作精神；',\n",
       " '具备IT基础知识，熟悉mysql、oracle、SqlServer之一，熟练操作linux操作系统，熟悉网络搭建及维护等；',\n",
       " '接受出差和加班。',\n",
       " '谢谢群主提供平台，五险一金，双休，有意者小窗。',\n",
       " '这个薪资还能招到人',\n",
       " '12K实施应该还可以吧',\n",
       " '有谁知道哪个版本兼容spark1.5.2的除了1.6',\n",
       " '使用hiveserver2命令启用hive是不是不用再启动metastore了，我从来没有启动这个，但各种操作都是正常的',\n",
       " '肖小和撤回了一条消息',\n",
       " '不用',\n",
       " '刘永领谢谢',\n",
       " '金思图像识别华为的产品吗',\n",
       " '不是报的错在你的项目路径下找不到类库么？你在你的开发工具中添加进去就可以了呀',\n",
       " '添加了不起作用啊',\n",
       " '总得都是linux下编译好的东西',\n",
       " '那应该还需要配置一个环境变量呀，现在记不起了',\n",
       " '我在码云上搜，怎么都搜不到什么',\n",
       " '你搜代码片段了啊',\n",
       " '我就是要搜代码片段啊',\n",
       " '项目这些也没有啊',\n",
       " '我做图像识别的',\n",
       " '你女朋友很漂亮啊',\n",
       " '还可以',\n",
       " '老婆和女友都很漂亮',\n",
       " '谁说你了',\n",
       " '通过xargs和管道来解决可以',\n",
       " '原来这样，谢谢大神',\n",
       " '其实新版本也都支持正则了',\n",
       " '不知道你那个是哪个版本？',\n",
       " '好像是根据本地目录去找hdfs上有没有',\n",
       " '感觉有的目录也不在我本地啊',\n",
       " '在在',\n",
       " '请问大家，我现在在做sparkstreaming+graphx计算，但仍然是打成jar在linux中执行脚本。有没有在用户场景下采用过REST服务在web界面中提交并获取结果的方案？',\n",
       " '我只需要知道这种REST是否已经有很多实践，或者指点几个名词之类的就好了',\n",
       " 'livy',\n",
       " '谢谢',\n",
       " '初学者问个问题hadoop的切片到底起了啥用处切片大小会影响mapper数量可是mapper具体读取的数据还是按照reader来的并不是取的切片内容那么划分切片干嘛呢望大佬解答谢谢',\n",
       " '切片数决定了有多少个mapperreader决定了每次执行map方法读取的数据比如默认一行',\n",
       " '切片越多并行度越高',\n",
       " '我好像理解了为什么会出现切片了,如果不进行切片处理的话一行一个mapper会起一堆mapper,或者一个文件一个mapper那么mapper计算压力太大.切片给每个mapper划分了一定范围,由于切片是可以指定的所以可能切片里会有跨界内容,所以切片只是个逻辑划分,只是框了一个大概范围.具体数据获取还是靠reader根据这一块切片范围进行均衡吧',\n",
       " '有人懂yarn吗',\n",
       " 'yarn',\n",
       " '有人懂吗',\n",
       " '来帮我解决一下问',\n",
       " 'abstract切片这块理解没问题reader这块听起来好像理解有点问题',\n",
       " '我再看看',\n",
       " '有谁在win下安装了hadoop环境',\n",
       " '看下这个地方是怎么配置的',\n",
       " '我这边格式化有这个问题',\n",
       " '你是在win上启动的吗..',\n",
       " '嗯',\n",
       " '本地想写远程hdfs',\n",
       " '有谁弄好的么发下配置文件看下去',\n",
       " '有人熟悉yarn吗真正在公司用过的',\n",
       " '按照网上一样写这样就可以了',\n",
       " '这目录问题是在哪里',\n",
       " 'win下有你这路径吗？',\n",
       " '盘符都没有',\n",
       " '有没有spark老司机有空？',\n",
       " '没有',\n",
       " '但是这么配置他能启动',\n",
       " '那我去spark社区吧问吧',\n",
       " 'spark什么问题',\n",
       " '刚刚打算看看Community怎么问问题的',\n",
       " '这个问题',\n",
       " '计数器？',\n",
       " '招java一名',\n",
       " '不是吧，',\n",
       " '就是一些常见的flatmap然后map',\n",
       " '也是偶然发现的bug，与RDD类型有关，还与uncacheTable的顺序有关',\n",
       " '所以应该问老司机级别的可能才会了解',\n",
       " '链接发出来看看？研究研究',\n",
       " '额，是我的有道云笔记咋发',\n",
       " '可以共享的',\n",
       " '发下复现问题的核心代码？',\n",
       " '这是复现问题的核心代码',\n",
       " '问题在于uncacheTable和mappartitionsRDD.flatmap的顺序',\n",
       " '有点长，其实就是一个临时里面读出个mappartitionsRDD然后uncacheTable，然后mappartitionsRDD.flatmap.toDF，再用sql就引发问题了',\n",
       " '虽然对结果没有任何影响',\n",
       " '新建完整类，以最少代码重现问题，发出来~',\n",
       " '请教个面试题，hive只能运行在hdfs上吗？',\n",
       " '好吧，，我整理下',\n",
       " '之前我也碰到一个无法解决的，换一种写法就可以',\n",
       " '运行时df.show报序列化问题，printscheme就不抱错~',\n",
       " '就这么多了，不能再少了，注释的是参考，换成parallelcollectionRDD或者改动uncacheTable到后面都能避开此问题',\n",
       " '我的加不加.rdd运行一样的',\n",
       " '不对',\n",
       " '这两天发言的人少了很多哦',\n",
       " '有人遇到过吗',\n",
       " '虚拟机的问题也来问哦',\n",
       " '文件日志为何不看一下？',\n",
       " 'sparkonyarn怎么调用第三方jar，比如说mongodb的驱动',\n",
       " '--jars',\n",
       " '谢谢，已解决',\n",
       " '还有一种就是平时要用的jar全放到hdfs上面去',\n",
       " '然后在spark-default中加入',\n",
       " '这版本冲突？',\n",
       " '字面意思是这样',\n",
       " '从2.3一路试下来到2.0版本了',\n",
       " '你虚拟机不2.3么',\n",
       " '虚拟机Ubuntu免密登陆必须是SU到超级管理员',\n",
       " '然后spark的用户组似乎不一样',\n",
       " '有连接问题',\n",
       " '然后懒得改了，直接win7上',\n",
       " '金思图像识别大哥很闲哦',\n",
       " '有点',\n",
       " '我压力比你们小',\n",
       " '工资还比你们高',\n",
       " '所以他经常出来嘲讽下你们',\n",
       " '激励你们',\n",
       " '房子车子老婆孩子人我都有',\n",
       " '可把你牛比坏了',\n",
       " '可把你牛比坏了',\n",
       " '可把你牛比坏了',\n",
       " '要像大佬学习',\n",
       " '已经把这个图片贴到他的空间了',\n",
       " '不是这个意思，我现在就算写代码估计没有人啊',\n",
       " '没有人要啊',\n",
       " '我也就差个房子/车子/老婆/孩子和人',\n",
       " '已经贴他空间了',\n",
       " '可把你牛比坏了',\n",
       " '要像大佬学习，就什么都有了，比如，房子，车子，老婆，孩子，人，票子',\n",
       " '大佬还有HIV',\n",
       " '你有没有',\n",
       " '听了一下午文贵说事',\n",
       " '感觉整个人都不好了',\n",
       " '增加dfs.replication为3，那么之前2备份的文件会增加一个备份吗',\n",
       " '郭文贵',\n",
       " '只会影响后面的',\n",
       " '健身教练最近还有消息么？',\n",
       " '慢慢看',\n",
       " '过得很逍遥',\n",
       " '收听敌台。自首吧',\n",
       " '最近还更新？没有被收买？',\n",
       " 'kafka+可以指定+消费数据的条数吗',\n",
       " '这个名字居然不是敏感词',\n",
       " '不是啊',\n",
       " '我用了很多年了',\n",
       " 'abstract有什么办法把之前的文件全备设置为3备份吗',\n",
       " '自己画地为牢，非要区分出敌我，都是炎黄子孙啊',\n",
       " '润之不是说你。懂得自然懂',\n",
       " '？',\n",
       " '其实我更想知道其三叔的复出，他是怎样的心。',\n",
       " '无邪...',\n",
       " '这种应该就是在画饼了吧',\n",
       " '万一老板意气相投的是加班这一项就很尴尬了',\n",
       " '你要多少了',\n",
       " '几年了？',\n",
       " '干的什么岗位啊',\n",
       " '本地运行sparkstreaming用到有状态保存算子updateStateByKey，需要设置checkpoint路径ssc.checkpoint(\"D:\\\\\\\\myTest\\\\\\\\ck\")',\n",
       " '能正常运行，就是会抛异常',\n",
       " 'php6年了',\n",
       " '项目经理岗',\n",
       " '武汉薪资应该挺高吧',\n",
       " 'PHP。。。',\n",
       " '武汉斗鱼啊',\n",
       " 'php的来学大数据么',\n",
       " '我去我有头衔了啊',\n",
       " '为啥有的没有',\n",
       " '斗鱼面试过',\n",
       " '有的有',\n",
       " '然后就挂了',\n",
       " '挂就挂了北',\n",
       " '挂个鱼去',\n",
       " '挂啥北啊',\n",
       " '斗鱼面试主播吗？',\n",
       " '直播造人',\n",
       " '你们好污啊',\n",
       " '卢老爷生前是个体面人',\n",
       " '本地运行sparkstreaming用到有状态保存算子updateStateByKey，需要设置checkpoint路径ssc.checkpoint(\"D:\\\\\\\\myTest\\\\\\\\ck\")',\n",
       " '能正常运行，就是会抛异常',\n",
       " '大数据开发人员，1年以上实战经验，国内项目，地点:青岛',\n",
       " '有感兴趣的小伙伴可以私聊我',\n",
       " '有人吗',\n",
       " '有',\n",
       " '为什么每次在家里xx.net就很难连上',\n",
       " '阿里云ECS服务器，有幸运券了：',\n",
       " '各位大神，请教个问题，为什么flume收集日志到hdfs，日志量会少很多，原始日志100多M，hdfs里面只有几十兆',\n",
       " '压缩了',\n",
       " '？',\n",
       " '没压缩，大小和行数都少了很多',\n",
       " '丢数据了呗',\n",
       " '恩，不知道哪里出了问题呢，看日志，没发现报错之类的',\n",
       " '你们hadoop生态的安全一般用什么来维护？',\n",
       " '用手',\n",
       " '对对',\n",
       " '真TM有道理',\n",
       " '两个RDD的两个关联操作有哪些算子啊，大神帮忙解答下',\n",
       " '大姜撤回了一条消息',\n",
       " '大姜撤回了一条消息',\n",
       " '没有自增主键的mysql数据怎么同步到hive中啊，需要做到实时或近实时',\n",
       " '这篇文章说的是对的',\n",
       " '嗯',\n",
       " '变更不变更自己不会测一下',\n",
       " '思考一下会不会变更',\n",
       " '老问个jj',\n",
       " 'web压力测试怎么cpu内存还很小网页就访问不了了啊？？',\n",
       " '有可能是什么原因',\n",
       " '根本就没访问成功',\n",
       " '老问个JJ',\n",
       " '你跟你妈说，今晚在同学家住',\n",
       " '那张图呢',\n",
       " '谁有那个图发一下',\n",
       " '酱爆肉',\n",
       " '武汉工资多高',\n",
       " '斗鱼有30k么',\n",
       " '斗鱼50K',\n",
       " '那还可以',\n",
       " '大数据搞基工程师',\n",
       " '冷少-包宿-葬爱家族我的微信里有',\n",
       " '怎么转过来',\n",
       " '微信保存到本地不就好了',\n",
       " '斗鱼有50k?',\n",
       " '什么50k',\n",
       " '卢老爷之前在斗鱼不是年薪千万吗',\n",
       " '卢本伟年薪千万',\n",
       " '每月百万',\n",
       " '卢老爷生前是个体面人',\n",
       " '啊呸',\n",
       " '下一位',\n",
       " '卢本伟牛逼',\n",
       " 'python中这种相对路径的问题怎么解决，有大神遇到过此类问题没',\n",
       " 'Snail撤回了一条消息',\n",
       " '甲：“做程序员太辛苦了，我想换行……我该怎么办？',\n",
       " '乙：“敲一下回车”',\n",
       " '感觉宝宝就是无根生的九奇技',\n",
       " '你们现在开发用scala还是py',\n",
       " 'py',\n",
       " '人生苦短，youneedpython',\n",
       " '我也感觉spark有意推py',\n",
       " '你用的几版？',\n",
       " '我目前用的2.7',\n",
       " '冷少-包宿-葬爱家族gay少出现',\n",
       " '葬爱',\n",
       " '深圳-扎心老铁撤回了一条消息',\n",
       " 'scala',\n",
       " 'scala',\n",
       " '大家能否认真研究一下这个文章',\n",
       " '你写的？',\n",
       " '秋天你上午说的问题，已经cache的数据不会变的',\n",
       " '除非你重新读一遍',\n",
       " '上海-刚子广播变量不叫cache吧，',\n",
       " '广州-spark-小白网上找的',\n",
       " '感觉这篇文章也不好',\n",
       " '更新的频率不好控制',\n",
       " '上海-刚子有没搞过在sparksteaming的driver端搞个定时线程去做一些事？',\n",
       " '广播变量应当也不会刷新数据',\n",
       " '需要你手动刷新',\n",
       " 'driver端做什么',\n",
       " '是，要有更新机制啊',\n",
       " 'Driver端搞个定时线程，线程获取数据，更新广播变量',\n",
       " '就像文章里说的',\n",
       " '文章里的解决方案是',\n",
       " '这是怎么回事？',\n",
       " '执行任务失败，数据插入到values__tmp__table__2',\n",
       " '这是哪里没设置对？',\n",
       " '打开http看下',\n",
       " '请问一下kettle定时，怎么避免数据重复呀',\n",
       " '问得太笼统，不知怎么答你。能想到的是把目标truncate。',\n",
       " '想的是任务每次执行前都把truncate掉，可是不知道怎么弄==',\n",
       " '可以在脚本INSERTOVERWRITETABLE覆盖就是了',\n",
       " 'drop了',\n",
       " '比如我现在想定时的把一个excel中的数据导入到数据库，而excel中的数据，每隔一段时间会增加一些，我只想再下次执行的时候增加新增的而不是所有数据全部再增加一次用sql的话我晓得怎么弄，kettle刚接触，不知道该怎么弄==',\n",
       " '还以为你问的是hive。浪费',\n",
       " '数据都在同一份excel里，你怎么区分哪些是新增的数据呢？',\n",
       " '加个字段代时间不就好了',\n",
       " '加个时间戳可以区分',\n",
       " '那不就完了',\n",
       " '可是怎么样才能不重复的添加进去==',\n",
       " '你新加的时间肯定比之前所有的时间新',\n",
       " '我现在定时就是无脑的导入--一直重复数据，难不成还要再把重复数据过滤嘛-',\n",
       " '取出中最大的来',\n",
       " '不就好了',\n",
       " '比较一下',\n",
       " '有谁用过百度地图地理位置爬虫的AK',\n",
       " 'date>max(date)',\n",
       " '写SQL我会我现在是想知道kettle里面怎么弄',\n",
       " 'kettle不也是写sql么',\n",
       " '我不会我不说了',\n",
       " 'kettle的excel输入没有字段值比较的功能',\n",
       " '我不知道在哪里能添加sql--',\n",
       " '不可以从excel取出来之后存到数据库的过程中比较？',\n",
       " '这样好麻烦==',\n",
       " '股价大涨了',\n",
       " '要么想把excel全量导入一张临时，再过滤时间字段，最后插入。',\n",
       " '哈哈中国降低汽车关税了买进口车',\n",
       " '这算是天朝赢了吗',\n",
       " '不是啊明显萎了啊',\n",
       " '不过这样也好比较美国一直经济低迷帮他们一下我们也好买进口车啊',\n",
       " '前几天还看新闻说要给BBA进口车加关税呢',\n",
       " '今天我们老大发话了啊说要降低汽车关税开放一些银行金融市场',\n",
       " '所以前几天的都作废了',\n",
       " '你很适合当天朝的喉舌，能把丧事当喜事办。',\n",
       " '我曹怎么怂了啊',\n",
       " '博鳌亚洲论坛习大大说了40多分钟',\n",
       " '这是要妥协吗',\n",
       " '说要扩大开放',\n",
       " '但是通信交通等重要行业是不开放的',\n",
       " '汽车无所谓啦',\n",
       " '股指大涨2个点多',\n",
       " '美帝才不会那么容易满足',\n",
       " '那我们就把大豆关税再取消了呗',\n",
       " '人家想制我们高科技。我们就必须妥协啊',\n",
       " '怎么这么能操心',\n",
       " 'WTO重要的承诺都没几条兑现',\n",
       " '那么之前说的500亿物品的关税?',\n",
       " '我买了股票我不操心。。。',\n",
       " '估计在下面改的吧',\n",
       " '在其位谋其职',\n",
       " '能增就能减啊',\n",
       " '呵呵哒',\n",
       " '没办法啊你想让欧美制裁我们吗',\n",
       " '你敢妥协一步美帝就敢再压迫一分',\n",
       " '股票最近幅度较大适量',\n",
       " '制裁了我们不就闭关锁国了？',\n",
       " '都妥协一下算了瞎TM折腾',\n",
       " '人地喺痛打流氓嗻',\n",
       " '美国主要是针对2025计划',\n",
       " '还要考虑这个只是烟雾弹实际的动作在后面或者已经开始了',\n",
       " '所以我们还得妥协啊比较科技是全球化的',\n",
       " '神仙打架，你们小鬼瞎操什么心',\n",
       " '我们知道的只是面的东西实际怎么样每人知道',\n",
       " '该干啥干啥',\n",
       " '唉和我们也有关啊',\n",
       " '各个都是人才',\n",
       " '广州-spark-小白你说了算吗？',\n",
       " '我要发出我的声音',\n",
       " '少折腾了',\n",
       " '那就继续做牲口，养肥等挨刀啦',\n",
       " '广州-spark-小白汪汪汪',\n",
       " '这不就是你的声音？',\n",
       " '瞎操心该发生的迟早要发生如果有做生意的需要根据政策去判断自己的走向',\n",
       " '贫民百姓就上班的上班吃饭的吃饭该吃吃该喝喝',\n",
       " 'C++引用参数不匹配',\n",
       " '建议把C++引用再研究一遍',\n",
       " '好的谢谢',\n",
       " 'C++17',\n",
       " 'forgotten您好，我是ros新手,问一下您现在截图中用的什么ide么',\n",
       " 'Qt',\n",
       " 'qt_creator?',\n",
       " '恩恩',\n",
       " '你安装一个插件',\n",
       " '好的，谢谢',\n",
       " '我现在只装了一个qt_creator',\n",
       " '能够运行基本的程序',\n",
       " '在装一个ros_qtc_plugin插件',\n",
       " 'https://ros-industrial.github.io/ros_qtc_plugin/_source/How-to-Install-Users.html',\n",
       " '这个插件是干什么用的，关联ros?',\n",
       " '按照这个装',\n",
       " '我现在是32位的unbuntu14.04',\n",
       " 'http://www.360doc.com/content/17/1121/20/48169514_705958629.shtml',\n",
       " '这个你也可以参考下',\n",
       " '没影响吧',\n",
       " '谢谢',\n",
       " '没事',\n",
       " '好的',\n",
       " '我按照这个步骤也是失败了，qt5.9也设置了qtchooser下的default.config还是装插件时提示qt版本低',\n",
       " '我想设置我的程序自启动，开机就全屏显示一张图片，于是写了一个service，但是启动的时候报错Gtk-WARNING**:cannotopendisplay:有哪位大侠遇到过吗？求教',\n",
       " 'Ros422串口总出现serial::serialException错误，节点死掉',\n",
       " '有遇到的吗',\n",
       " 'http://mc.dfrobot.com.cn/forum.php?mod=viewthread&tid=36541&extra=这是我写的一个关于固态雷达的论坛，欢迎在评论区吐槽[捂脸][捂脸][捂脸]',\n",
       " '价格怎样',\n",
       " '公司给买的，好像量多比较便宜',\n",
       " '请问搭建Hadoop集群，要在主节点和其余全部辅助节点全部配置hadoop-env.sh、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml',\n",
       " '吗？',\n",
       " '对头，所有节点配置得一样',\n",
       " '还是只要主节点配置这五个文件，其余节点不用？',\n",
       " '噢噢噢',\n",
       " '比如，我五台机器，所有操作全部要一致对吧？',\n",
       " '嗯，你可以先配置好一台，然后直接复制',\n",
       " '我是新手，实习生，请群里面大神们多多指教啊',\n",
       " 'SCP传过去是吗？',\n",
       " '都可以，',\n",
       " '我都不太会',\n",
       " '请问您那里有详细点的操作步骤吗？',\n",
       " '我已经搭了两周了',\n",
       " '哎',\n",
       " '没写笔记，',\n",
       " '那我先试着继续弄吧',\n",
       " '网上教程不是很多吗？好像很多都可以用的',\n",
       " '不大全',\n",
       " '没那么详细',\n",
       " '我因为是新手，很多东西都不懂',\n",
       " 'http://www.tianshouzhi.com/api/tutorials/hadoop',\n",
       " '这个教程只有伪分布式集群，很多东西也没讲到，不过你可以先学习一下，后续的自己慢慢研究，网上找资料',\n",
       " '恩那',\n",
       " '谢谢你',\n",
       " '我先看看',\n",
       " '蚂蚁金服19级实习招人啦！需要内推的可以发到这个邮箱：wesley.hwwantfin.com',\n",
       " '19级？',\n",
       " '19届',\n",
       " '的意思吧',\n",
       " '明年毕业的',\n",
       " '蚂蚁金服19年应届生实习招人啦！需要内推的可以发到这个邮箱：wesley.hwwantfin.com。待遇高于其他公司，有巨大的成长空间，大牛云集，我们是支付宝事业部的，不管你会java还是算法，只要你自信，就来吧',\n",
       " '工作地点在北京吗？',\n",
       " '杭州',\n",
       " '谢谢',\n",
       " '大佬们，我使用spark-submit提交到yarn-cluster,报这个错误，要怎么解决啊？',\n",
       " '你看看hdfs的目录上是否有这个文件。报文件找不到',\n",
       " '是我app的问题，我设置成了spark://master:7077,但是我spark-submit是设置的yarn-cluster，所以没有上传这个文件',\n",
       " '有没有哪位大佬遇到过，container被强制kill掉的况，Exitcodeis143，，，好像是由于MAPREDUCE–5645的原因，该怎么解决呢？',\n",
       " '谁有NppTP64位的插件给发一个，谢谢',\n",
       " '大家好，我是丶_____日久厌。来自广东佛山的射手座男一枚~',\n",
       " '这个我已经禁止了防火墙，可是还是不解决，求助大神们',\n",
       " '你ping完后执行arp-a看看ip是否解析成mac地址',\n",
       " '大佬们，我想问个Zookeeper的问题',\n",
       " '我zookeeper创建了一个znode，通过addauthdigestusername：pswd创建了一个用户，通过setAclpathauth:username:passwd:cdrwa设置了权限，但是我现在发现我不能查看这个znode了，也不能删除，权限也不能重新设置。总是报这个错',\n",
       " '不知道这个问题怎么解决啊？',\n",
       " '这是znode的权限',\n",
       " '但是我发现我现在不能操作这个节点了，这个怎么办',\n",
       " '实在不行看到data目录重启。这样会重建基础库。前提是没什么重要数据。',\n",
       " '干掉',\n",
       " '重启虽然能解决问题，但是万一以后还遇到这个问题咋办，',\n",
       " '现在我是删不掉这个节点，也看不了这个节点，ls都报错，然后直接退出客户端',\n",
       " '================================================================',\n",
       " '消息分组:已退出的群',\n",
       " '================================================================',\n",
       " '消息对象:JHipster中文社区||',\n",
       " '================================================================',\n",
       " '那Nodejs呢',\n",
       " 'js和node类似java和jvm',\n",
       " 'node是在v8基础上的',\n",
       " '有谁对接过自身实现的oauth，怎么实现的？我是小白，有思路的也可以提提急求',\n",
       " 'springcloudconfig的controller把静态资源也拦截了',\n",
       " '你的意思是写个拦截器把所有controller的请求都拦截了然后在拦截器中实现对oauth的权限受理？？？',\n",
       " '有人用vue写过jhipster的ui吗',\n",
       " '是不是像这种js文件都是写在.ts文件中的',\n",
       " '现在要修改一些页面上的东西加一个ajax请求',\n",
       " '有没有案例给欣赏下的大佬们',\n",
       " '做成前后端分离的就行了',\n",
       " '就是那个生成的页面',\n",
       " '目录是这样的',\n",
       " '怎么加js',\n",
       " '加在html中页面就不显示了',\n",
       " '加在ts中找不到方法',\n",
       " 'ts加jquery的依赖',\n",
       " '直接用js的写法',\n",
       " '引入jquery?在ts中？',\n",
       " '我加在ts中的js每次访问都是找不到方法',\n",
       " '引入jquery报错有人遇到过么',\n",
       " '一丝尘埃加入本群。',\n",
       " '-恆-加入本群。',\n",
       " 'SHERMAN加入本群。',\n",
       " '你爹临死前加入本群。',\n",
       " 'yarninstall',\n",
       " '请教大伙儿，jhipster使用uaa和gateway方式创建微服务，使用postman测试增删改查，出现这种况，怎么解决？',\n",
       " '注册中心启动空白页面怎么回事啊',\n",
       " '我的也是。',\n",
       " 'gateway能看。。',\n",
       " 'https://blog.csdn.net/mr_rain/article/details/72770527',\n",
       " '谢谢我试试哈',\n",
       " '要登录',\n",
       " 'gateway登录然后再请求',\n",
       " '我的问题么？',\n",
       " '网关已经登录了',\n",
       " '请求是用http工具还是当前网页',\n",
       " '畫心。谢谢可以了',\n",
       " '恩',\n",
       " '我的问题还没解决。。',\n",
       " 'postman没登录',\n",
       " '我试试',\n",
       " 'https://stackoverflow.com/questions/27182701/how-do-i-send-spring-csrf-token-from-postman-rest-client',\n",
       " '用的什么字体',\n",
       " '没看明白。。。',\n",
       " '那个js方法放哪里能行',\n",
       " '直接post用户名密码会报错缺少头部csrftoken',\n",
       " '所以应该在环境出配置，我没明白的是环境这块怎么加那个token',\n",
       " '你看下cookie',\n",
       " '看到了，',\n",
       " '这样?',\n",
       " '估计不行',\n",
       " '这个登录方法写在哪里的？',\n",
       " '这个xsrf是浏览器的不是postman的',\n",
       " '不是',\n",
       " '客户端的',\n",
       " '刚才的登录方法是写在chromeconsole中的？',\n",
       " '不是postman的environment',\n",
       " '我折腾到这块还没弄呢',\n",
       " '我这卡这了。。',\n",
       " '前后端分离的前端用vue请求用axios这个支持还没弄呢',\n",
       " '我在找找办法',\n",
       " '应该是先登录注册中心',\n",
       " '然后再请求就好了',\n",
       " '请求的是gateway的login',\n",
       " '请求account获取登录信息如果没的话就没登录跳转登录login还有个logout的接口',\n",
       " '我这个是uaa进行鉴权登录的，gateway应该是有权限的吧',\n",
       " 'gateway也是走的uaa',\n",
       " '对',\n",
       " '想问下，有没有jhipster结合vue的框架代码',\n",
       " '前端大家都是怎么改的',\n",
       " '用的什么',\n",
       " 'Bobby九五之尊',\n",
       " '...',\n",
       " 'Demon加入本群。',\n",
       " '土豆青年加入本群。',\n",
       " '求个支持react正式版发布的确切时间。',\n",
       " '更关心vue版',\n",
       " 'JHipster哪里有中文教程',\n",
       " '？',\n",
       " '用谷歌自带的那个翻译的吗楼上',\n",
       " '是的',\n",
       " '神经网络翻译就是这么强大。',\n",
       " 'jhipster好像找不到vue版的ui',\n",
       " '天缘加入本群。',\n",
       " '瑜在吗？',\n",
       " 'jhipster生成的demo，使用uaa鉴权，gateway，这边刚创建了实体，可以查，但是删除与添加修改都报这样一个错误，有人知道么？',\n",
       " '你那个401的问题解决了吗',\n",
       " '解决了',\n",
       " '用的插件postman',\n",
       " '我发现我们都刚入坑啊',\n",
       " '我也遇到了',\n",
       " '什么插件呢',\n",
       " '这两个，然后在网关登录下',\n",
       " '你的postman是谷歌里面的是吧',\n",
       " '我下载的是单独的一个应用',\n",
       " '是',\n",
       " '单独应用我也下了',\n",
       " '不行',\n",
       " '奥那我再谷歌里面装上哈',\n",
       " '估计你下个问题我快遇到了哈哈哈',\n",
       " '同样一个链接，',\n",
       " '只是一个在浏览器中登录过',\n",
       " '我去这么坑啊',\n",
       " '客户端的失败，',\n",
       " '我估计是谷歌浏览器帮我们做了什么',\n",
       " '所以成功了',\n",
       " '我还发现好像restful方式也不是太行',\n",
       " '不是谷歌，可能是因为浏览器可以执行登录，然后这个登录的token就写在了浏览器中',\n",
       " '奥缓存了吗',\n",
       " '感觉应该是',\n",
       " '我去试试坑哈',\n",
       " '前两个啊',\n",
       " '对',\n",
       " '好的',\n",
       " 'npminstall的时候。。。',\n",
       " '拦截器记得开',\n",
       " '没有人用VScode的restclient吗?',\n",
       " '怎么解决大佬们',\n",
       " 'ok',\n",
       " '我用的yarn，前端用node-sass',\n",
       " '嗯我也用的yarn',\n",
       " '是在回答我么',\n",
       " '只看到这个，你不试试？',\n",
       " '问题不在这儿其他项目npminstall都ok。这个就是jhip生成的这样就GG',\n",
       " 'yarn对于typescript需要做冲突选择。',\n",
       " 'npm这方面稍微傻瓜点，不需要自己去选版本。',\n",
       " '马上我也要再配这个登录方法了',\n",
       " 'csrfdisable',\n",
       " '在找这个类呢，找一会了',\n",
       " '见鬼了，。。',\n",
       " '不行么',\n",
       " '我这文件里默认就有这一块',\n",
       " '要不然看下uaa的吧',\n",
       " '这就是uaa的。。',\n",
       " '你直接post呢',\n",
       " '只能get',\n",
       " 'post不行，问题肯定在springsecurity了',\n",
       " 'restful只能get。。',\n",
       " '通过网关访问的么？',\n",
       " '通过网关，网关去uaa鉴权',\n",
       " '网关里',\n",
       " '加上就好使了',\n",
       " '在gateway里面',\n",
       " 'gateway中没找到这个，',\n",
       " '只有在uaa有',\n",
       " '难道我这是直接访问uaa进行鉴权的？',\n",
       " '发现了，我试试',\n",
       " '刚才组长来了',\n",
       " '看我这么改说了我一顿。。',\n",
       " '让我addMatchers',\n",
       " 'permitAll',\n",
       " '效果怎么样？',\n",
       " '正在搞，稍等',\n",
       " 'springsecurity相比shiro到底怎么样？请大家说说',\n",
       " '复杂',\n",
       " '以前的acegi，比shiro复杂多了',\n",
       " 'shiro是简单些唉',\n",
       " '但是总想着spring全家桶。。。有一样的吗',\n",
       " '我改的有问题么？怎么还是不行',\n",
       " 'acegi太晦涩了，不知道spring收了现在怎么样',\n",
       " '畫心。可能要加到签名',\n",
       " '前面',\n",
       " '这玩意前后顺序上不同效果不一样',\n",
       " '网关给微服务映射了一个路径，那么我放开这个路径是对的么？',\n",
       " '绝择按照你的方法的确可以了，谢谢，明天我再问问组长他们是怎么搞的吧',\n",
       " '可能我又弄错了',\n",
       " 'http://url.cn/5hTr4jU?_wv=41729&_wvx=10',\n",
       " '各位vue用什么编辑器比较好用啊？我用eclipse编辑着太别扭了',\n",
       " 'webstorm',\n",
       " '再安装一个vue的扩展',\n",
       " '美滋滋',\n",
       " 'txt才是最好的！',\n",
       " 'txt啊？学习用它应该不错，工作用着给怕速度跟不上啊',\n",
       " '谢谢',\n",
       " 'vscode吧vim竟然没人推荐',\n",
       " 'vscode和webstorm那个更好用呢？',\n",
       " 'vscode',\n",
       " 'emacs不也没人推荐么',\n",
       " 'sumbline和vscode都可以吧',\n",
       " '嗯嗯好的谢谢各位我用vscode吧',\n",
       " 'hbuilder',\n",
       " 'idea',\n",
       " 'MUI',\n",
       " 'hbuilder是eclipse做的吧',\n",
       " 'idea',\n",
       " '不是吧。。',\n",
       " '万能idea',\n",
       " 'hbuilder看着像eclipse',\n",
       " 'ideaeclipse都耗资源，能不能出个轻量的',\n",
       " '万能txt!',\n",
       " 'hbuilder里面很多东西和eclipse类似的不过运行起有点卡',\n",
       " '那只能vim了，，',\n",
       " '炒鸡轻',\n",
       " 'overstackflow上排名第一的问题是vim怎么退出/斜眼笑',\n",
       " '：wq',\n",
       " 'jhipster项目，接收客户端请求的是网关而不是直接微服务对吧？',\n",
       " '。。。这个如果你请求网关就是网关',\n",
       " '嗯发现跨域设置没用',\n",
       " '请求微服务就是为服务',\n",
       " '应该是网关再去调服务',\n",
       " '道理应该是这样。',\n",
       " '调试可以直接去服务',\n",
       " '我的想法，至于那个如何互相信任的，没想明白，可能是带token了',\n",
       " '吃饭去',\n",
       " '瑜在吗？',\n",
       " '？',\n",
       " '怎么了',\n",
       " '上次你的那个401是怎么解决的？',\n",
       " '注册中心的',\n",
       " '瑜上次注册中心那个401的问题，是怎么解决的啊？',\n",
       " '401？',\n",
       " '有图么',\n",
       " '你之前不是发了一张401的问题',\n",
       " '不记得了',\n",
       " '我给你找找',\n",
       " '像这个样子的',\n",
       " 'uaa没编译好？',\n",
       " '不是',\n",
       " '401的状态码，意思应该是没有授权吧',\n",
       " '没见过',\n",
       " '这个我把它的passwordencoder屏蔽了',\n",
       " '然后密码变成明文就可以了',\n",
       " '这个路径还是出不来不过uaa可以了',\n",
       " 'passwordencoder屏蔽了，你是在那屏蔽的啊？',\n",
       " '有没有代码截图什么的？',\n",
       " 'jhi_persistent_audit_event这个是什么作用？',\n",
       " '你画的这个红框是什么意思啊？',\n",
       " '我的也有这个',\n",
       " '各位java骑共享单车还要压金？包月贵？用这个吧！免压金！免押金骑所有品牌单车，来领取免费骑行http://m.qnche.com/h5/redPacketShare.html?inviteCode=a40178839057',\n",
       " '这种聚合的迟早被封',\n",
       " '然后就卖用户。',\n",
       " '分久必合合久必分',\n",
       " '赢/am加入本群。',\n",
       " '有朋友知道jhipster中swagger的json文件怎么获取吗',\n",
       " '蛋蛋加入本群。',\n",
       " '茂茂加入本群。',\n",
       " '9527加入本群。',\n",
       " '好久不见，你还好吗？加入本群。',\n",
       " 'jhipster前端用ng写ajax有人写过么',\n",
       " '没用过ng求个参考',\n",
       " 'http://localhost:8080/v2/api-docsswaggerdejson',\n",
       " 'YuQ加入本群。',\n",
       " '没有昵称加入本群。',\n",
       " '水清木华加入本群。',\n",
       " '吴先生撤回了一条消息',\n",
       " \"有没有人知道引入xml文件报错Causedby:org.xml.sax.SAXParseException:cvc-elt.1:找不到元素'settings'的声明。\",\n",
       " '大神们帮帮忙困这里一天了',\n",
       " '加头约束',\n",
       " '什么意思啊',\n",
       " '这个xml是对接别的项目的后台配置给我就这样',\n",
       " '爱莫能助，我看错群了',\n",
       " '$scope这个报错的原因谁知道呀',\n",
       " '看下ng文档吧',\n",
       " 'JHipster的前端不是ng的么',\n",
       " '星晓加入本群。',\n",
       " 'Amoni加入本群。',\n",
       " '大佬们，现在我生成了一个单体应用也跑起来了，为啥访问地址是空白页面呢，yarnstart也执行过了',\n",
       " 'yarnstart不能关闭',\n",
       " '我没有关闭，yarnstart执行之后自动弹出了一个页面',\n",
       " '然后是空白页？',\n",
       " 'ctrl+c',\n",
       " '注册中心么？',\n",
       " '这是生成的那个页面',\n",
       " '看地址栏里面的地址',\n",
       " '网关？',\n",
       " '我的是8080',\n",
       " '这个不是网关',\n",
       " '就是一个单体应用',\n",
       " '那你这不是启动了吗？',\n",
       " '为什么说是空白页？',\n",
       " '我没有启动项目，这是执行yarnstart之后自动跳出来的',\n",
       " '这是访问启动之后的项目，就是空白页',\n",
       " '这算是前后端分离嘛',\n",
       " '前端页面还自动有一个端口吗',\n",
       " '你要打成war包，然后运行war包就不会空白了',\n",
       " 'dev模式是分开的',\n",
       " '微服务应用我没访问过页面，都是看网关的api',\n",
       " '他这不是微服务',\n",
       " '检查一下你的index.html，有时候出错了里面是空的',\n",
       " '不知，刚学不久',\n",
       " 'index页面确实是空的',\n",
       " '内容全被注释掉了',\n",
       " '软件测试能力是指的什么，有谁有写过类似投标文件吗',\n",
       " '没人做页面的ajax或者别的修改什么的么',\n",
       " '后端的多点吧',\n",
       " '后端简单springboot的',\n",
       " '前端好难搞',\n",
       " '￠約顁々请问一下执行yarnstart',\n",
       " '之后，出现了三个端口号，这是啥啊',\n",
       " '没注意看过，我也是刚刚研究',\n",
       " '好吧',\n",
       " '================================================================',\n",
       " '消息分组:已退出的群',\n",
       " '================================================================',\n",
       " '消息对象:dx',\n",
       " '================================================================',\n",
       " ...]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from chatterbot import ChatBot\n",
    "# from chatterbot.trainers import ListTrainer\n",
    "# Chinese_bot = ChatBot(\"Training\")\n",
    "# Chinese_bot.set_trainer(ListTrainer)\n",
    "# Chinese_bot.train(sec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seq2Seq 属于 Encoder-Decoder 结构，这里看看常见的 Encoder-Decoder 结构。基本思想就是利用两个 RNN，一个 RNN 作为 Encoder，另一个 RNN 作为 Decoder。Encoder 负责将输入序列压缩成指定长度的向量，这个向量就可以看成是这个序列的语义，这个过程称为编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/Users/zhangjianfeng/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, LSTM, Dense\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Batch size 的大小\n",
    "batch_size = 32  \n",
    "# 迭代次数epochs\n",
    "# epochs = 100\n",
    "epochs = 10\n",
    "# 编码空间的维度Latent dimensionality \n",
    "latent_dim = 256  \n",
    "# 要训练的样本数\n",
    "# num_samples = 5000 \n",
    "num_samples = 500\n",
    "#设置语料的路径\n",
    "data_path = '../data/13/files.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples: 500\n",
      "Number of unique input tokens: 716\n",
      "Number of unique output tokens: 1038\n",
      "Max sequence length for inputs: 40\n",
      "Max sequence length for outputs: 36\n"
     ]
    }
   ],
   "source": [
    "# 把语料向量化：\n",
    "#把数据向量话\n",
    "input_texts = []\n",
    "target_texts = []\n",
    "input_characters = set()\n",
    "target_characters = set()\n",
    "\n",
    "with open(data_path, 'r', encoding='utf-8') as f:\n",
    "    lines = f.read().split('\\n')\n",
    "for line in lines[: min(num_samples, len(lines) - 1)]:\n",
    "    #print(line)\n",
    "    input_text, target_text = line.split('\\t')\n",
    "    # We use \"tab\" as the \"start sequence\" character\n",
    "    # for the targets, and \"\\n\" as \"end sequence\" character.\n",
    "    target_text = target_text[0:100]\n",
    "    target_text = '\\t' + target_text + '\\n'\n",
    "    input_texts.append(input_text)\n",
    "    target_texts.append(target_text)\n",
    "\n",
    "    for char in input_text:\n",
    "        if char not in input_characters:\n",
    "            input_characters.add(char)\n",
    "    for char in target_text:\n",
    "        if char not in target_characters:\n",
    "            target_characters.add(char)\n",
    "\n",
    "input_characters = sorted(list(input_characters))\n",
    "target_characters = sorted(list(target_characters))\n",
    "num_encoder_tokens = len(input_characters)\n",
    "num_decoder_tokens = len(target_characters)\n",
    "max_encoder_seq_length = max([len(txt) for txt in input_texts])\n",
    "max_decoder_seq_length = max([len(txt) for txt in target_texts])\n",
    "\n",
    "print('Number of samples:', len(input_texts))\n",
    "print('Number of unique input tokens:', num_encoder_tokens)\n",
    "print('Number of unique output tokens:', num_decoder_tokens)\n",
    "print('Max sequence length for inputs:', max_encoder_seq_length)\n",
    "print('Max sequence length for outputs:', max_decoder_seq_length)\n",
    "\n",
    "input_token_index = dict(\n",
    "    [(char, i) for i, char in enumerate(input_characters)])\n",
    "target_token_index = dict(\n",
    "    [(char, i) for i, char in enumerate(target_characters)])\n",
    "\n",
    "encoder_input_data = np.zeros(\n",
    "    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),dtype='float32')\n",
    "decoder_input_data = np.zeros(\n",
    "    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),dtype='float32')\n",
    "decoder_target_data = np.zeros(\n",
    "    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),dtype='float32')\n",
    "\n",
    "for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):\n",
    "    for t, char in enumerate(input_text):\n",
    "        encoder_input_data[i, t, input_token_index[char]] = 1.\n",
    "    for t, char in enumerate(target_text):\n",
    "        # decoder_target_data is ahead of decoder_input_data by one timestep\n",
    "        decoder_input_data[i, t, target_token_index[char]] = 1.\n",
    "        if t > 0:\n",
    "            # decoder_target_data will be ahead by one timestep\n",
    "            # and will not include the start character.\n",
    "            decoder_target_data[i, t - 1, target_token_index[char]] = 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 400 samples, validate on 100 samples\n",
      "Epoch 1/10\n",
      "400/400 [==============================] - 5s 14ms/step - loss: 3.1284 - val_loss: 2.7257\n",
      "Epoch 2/10\n",
      "400/400 [==============================] - 5s 12ms/step - loss: 2.9159 - val_loss: 2.7183\n",
      "Epoch 3/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.8912 - val_loss: 2.7224\n",
      "Epoch 4/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.8760 - val_loss: 2.7331\n",
      "Epoch 5/10\n",
      "400/400 [==============================] - 6s 14ms/step - loss: 2.8687 - val_loss: 2.7478\n",
      "Epoch 6/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.8478 - val_loss: 2.7435\n",
      "Epoch 7/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.8319 - val_loss: 2.7525\n",
      "Epoch 8/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.8315 - val_loss: 2.7671\n",
      "Epoch 9/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.8084 - val_loss: 2.7746\n",
      "Epoch 10/10\n",
      "400/400 [==============================] - 5s 13ms/step - loss: 2.7917 - val_loss: 2.7927\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/zhangjianfeng/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2344: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1/while/Exit_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'lstm_1/while/Exit_3:0' shape=(?, 256) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).\n",
      "  str(node.arguments) + '. They will not be included '\n"
     ]
    }
   ],
   "source": [
    "encoder_inputs = Input(shape=(None, num_encoder_tokens))\n",
    "encoder = LSTM(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_inputs)\n",
    "# 输出 `encoder_outputs` \n",
    "encoder_states = [state_h, state_c]\n",
    "\n",
    "# 状态 `encoder_states` \n",
    "decoder_inputs = Input(shape=(None, num_decoder_tokens))\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_outputs, _, _ = decoder_lstm(decoder_inputs,\n",
    "                       initial_state=encoder_states)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "\n",
    "# 定义模型\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "\n",
    "# 训练\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')\n",
    "model.fit([encoder_input_data, decoder_input_data], decoder_target_data,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_split=0.2)\n",
    "# 保存模型\n",
    "model.save('s2s.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_model = Model(encoder_inputs, encoder_states)\n",
    "\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
    "decoder_outputs, state_h, state_c = decoder_lstm(\n",
    "    decoder_inputs, initial_state=decoder_states_inputs)\n",
    "decoder_states = [state_h, state_c]\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + decoder_states_inputs,\n",
    "    [decoder_outputs] + decoder_states)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reverse_input_char_index = dict(\n",
    "    (i, char) for char, i in input_token_index.items())\n",
    "reverse_target_char_index = dict(\n",
    "    (i, char) for char, i in target_token_index.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    states_value = encoder_model.predict(input_seq)\n",
    "    #print(states_value)\n",
    "\n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "    # Populate the first character of target sequence with the start character.\n",
    "    target_seq[0, 0, target_token_index['\\t']] = 1.\n",
    "\n",
    "    # Sampling loop for a batch of sequences\n",
    "    # (to simplify, here we assume a batch of size 1).\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "        output_tokens, h, c = decoder_model.predict(\n",
    "            [target_seq] + states_value)\n",
    "\n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_char = reverse_target_char_index[sampled_token_index]\n",
    "        decoded_sentence += sampled_char\n",
    "        if (sampled_char == '\\n' or\n",
    "           len(decoded_sentence) > max_decoder_seq_length):\n",
    "            stop_condition = True\n",
    "\n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "        target_seq[0, 0, sampled_token_index] = 1.\n",
    "        # 更新状态\n",
    "        states_value = [h, c]\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_ans(question):\n",
    "        inseq = np.zeros((len(question), max_encoder_seq_length, num_encoder_tokens),dtype='float16')\n",
    "        decoded_sentence = decode_sequence(inseq)\n",
    "        return decoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decoded sentence: 油油油油油油油油油油油油油油油\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Decoded sentence:', predict_ans(\"挖机履带掉了怎么装上去\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
